IN THE CLAIMS; 



Please amend Claims 1, 5. 7, 8 and 11. 
Please add Claims 14-16. 



CO 



1. (Currently Amend^) A An image processing method for recovery of r e cov e ring 3D a 
scene structure and camera motion from successive image data obtain e d from a multi imag e 
s e qu e nc e , wh e r e in a r e f e r e rro e imag e of th e s e qu e nc e is tak e n by a cam e ra at a r e fer e nc e 
p e rsp e ctiv e and on e or mor e succ e ssiv e imag e s of th e s e qu e nc e ar e tak e n at on e or mor e 
succ e ssiv e diff e r e nt p e rsp e ctives by translating and/or rotating th e cam e ra, th e m e thod 
comprising th e s t e ps of : 

(a) determini ng compan ig the image data of a reference shifts for e ach succ e ssiv e 
image to the image data of the s iccessive image, the successive image being taken by 
translating or rotating the cameraw^h respect to the reference image and the image data being 
one or more characteristics selected from a group consisting of points, lines and intensities 
with r e sp e ct to th e r e f e r e nc e imag e ; th e shifts b e ing d e riv e d from th e cam e ra translation 



and/or rotation from th e r e fer e nc e p e rsp e ctive to th e succ e ssiv e diff e r e nt p e rspectiv e s ; 

(b) constructing a detemnining the image shift data shift for the successive matrix that 
incorporat e s th e image with r espect to the reference data shifts for e ach image; 

(c) calculating two rank 3 factor matrices from th e shift data matrix using SVD, on e 
rank 3 factor matrix constructing shift data representation that incorporate the image data 
shifts for each image corresponding to the 3D structure and th e other rank 3 factor matrix 
corresponding to the earner! . motion; and 
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(d) r e cov e ring determining the direction of the camera motion and the 3D structure 
from the shift data representation. 3D structur e matrix by solving a linear e quation; and 

( e ) r e cov e ring th e carqgra motion from th e cam e ra motion matrix using th e r e cover e d 
3D structur e . 

2. (Original) The method of claiiA 1 , wherein the image data is one or more selected from 
the group consisting of points, lines and intensities. 



3. (Currently Amended) The methoa of claim 1, wherein the step of determining the image 
data shifts includes initially recovering and compensating for camera rotation. 

4. (Original) The method of claim , wherein step (b) comprises: 



computing H and D CH , wheije H is a (N tot - 3)x N tot matrix and D CH = C' m D H T 

and Mot = 2N P + 2 N L + where N p , Nl, and TV* equal the number of points, lines and 
intensities, respectively, and C is li(N\ -1) x (N\ -1) matrix with C ir = <5, r + 1 where C, r is a 
constant table of values for all images and<5, r is an operator equal to 1 if / = /' and 0 if / * /' , 



and D = [S co L A co x A], H t is th| transpose of the H matrix, where H is a matrix defined such 
that H T is an identity matrix and annihilates V x , *F ,% where s fpj <s> L Y£ co^l ] and 



similarly for the y and z components where, co { and co L are constant weights that the user sets, 

|~{P„ -(zxA)}l 



and where ^ = 



{P^txxA 



^•(yxA)}" 
.{p^(yxA)}. 
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unit normal to the plane combining the line and the camera center at the reference image and 
where x, y, z are unit vectors ia the x, y and z directions, and where 



X 

„(1) 



to 
to 



¥ = 



tracked point in the reference imagfe and 




where #=(x,)>) is the image position of the 



the three point rotational flows r 



5 > ; ) 5 A " (2) ( Jc j>')> r(3) ( JC '> ; ) are defined by 



anil where 



l},^ s -{w -r^(p)}, and where /? gives the image 



coordinates of the pixel positions, and 

where A is an N f x 2N L matrix mere each row corresponds to a different image i and equals 

[{Py • SA 1 ) T \p l - SA 1 f J where Jnd P L are unit 3-vectors projecting onto two directions 
A, x (f x A, )and z x A, which w| refer to respectively as the upper and lower directions, 
where SA 1 is the line flow 5A J § Aj - A? , and f is the unit vector in the z direction and 
where S is a JV, x 2 AT p matrix Adhere each row corresponds to a different image i and equals 

[{^ } r {s^, } r J , where s* m = q* m -j q° m and denotes the image displacement for the /n-th tracked 
point and where A is a N r x M x matrix, where each row corresponds to an image i and equals 
{a/' ) T where AI is the change in image intensity with respect to the reference image and 
where /'" denotes the /-th intensity image and where V n = V (p„ ) denotes the image intensity 
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< 



at the H-th pixel position in \ and where the notation { V) is used to denote a vector with 
elements given by the V a 

5. (Currently Amended) The method of claim 1, wherein step (c) comprises: 

computing a^best rank-3|factorization of D CH « where are rank 3 

matrices corresponding respectively to motion and structure, using SVD. 

6. (Original) The method of claim 1, wherein step (d) comprises: 

eliminating structure unknowrfs Q z , B z and Z" 1 from the O a to get 3N tot linear constraints 
on the U and Q using the linear equation, [o x ® y ® z ] = H T S {2) U + [¥ x W y % ]□ , where t/ 



and Q are unknown 3x3 mai 



flow vectors respectively, and solving these constraints with O(N t0t ) computations using the 



inear equation, \& x Q? y 
Ltrices, and O and *F r< 



epresent total translational and rotational 



SVD, where the total translational flow vectors are defined by O x = 



for the y and z components, and 



t^ts, 



and similarly 



/ . 

e/ce 



.{4 



where A is the normal to a first plane that 



passes through the/center of projection and the imaged line in the reference image, and the 
normal to a second plane, B is defined by requiring B • A = 0 and B ■ Q = -1 for any point Q 
the 3D line L, anJ where B u ^BP U and B L = B • ? L are the upper and lower components of 
B, and where 
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and where 

0^-{z-%\0 fy ^-{z-\}^ k s {z-'(V/.p)}, and 

where 0 is the 3d coordinate for a 3D tracked point corresponding to an image pixel in 
the reference image, and Z n h the depth from the camera to the 3D point imaged at the n-th 
pixel along the cameras optic^ axis; and 

recovering the structurl unknowns Q z , B z and Z" 1 from 
O x oJ=H r 5 ( % + [^ x 4|¥ 2 ]Q, given f/andn. 



7. (Currently Amended) Thl method of claim 1 , wherein step (d) further (e) comprises: 
using S®U « [<D X Oioj and 

D CH « C"^ {7; }0 r H T + C'f {7; )o JH T + C~% {T z }<D[H t to recover the translations, and 
recovering the rotations c^.co^co^ from 



<oy xn + «J= C"^d; -(c %> r + + , wherein C 

constant (Ni-1) x (Ni-M) matrix with C, r = <5, r + 1 and T represents the translation. 



is a 



8 
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8. (Currently Amended) A method for r e covering 3D An image processing method for 
recovery of a scene structure andVamera motion from successive image data obtained from a 
multi-image sequence , wh e r e in a r e fer e nc e imag e of th e s e qu e nc e is tak e n by a cam e ra at a 
r e f e r e nc e p e rsp e ctiv e and on e or mote succ e ssiv e imag e s of th e s e qu e nc e or e tak e n at on e or 
mor e succ e ssiv e diff e r e nt persp e ctiv e ^ by translating and/or rotating th e cam e ra, th e m e thod 
comprising th e st e p s of : 

(a) initially r e cov e ring and comp e nsating for cam e ra rotation aft e r warping th e 
comparing a reference image to successive images, wherein the successive image is taken by 
translating or rotating the camera by tn e pr e viously e stimat e d translationol displacem e nts ; 

(b) d e t e rmining imag e data shots for e ach succ e ssiv e imag e with r e sp e ct to th e 
r e f e r e nc e imag e ; th e shifts b e ing d e riv e d from th e camera translation and/or rotation from th e 
r e f e r e nc e p e rsp e ctiv e to th e succ e ssiv e diff e r e nt p e rsp e ctiv e s compensating for camera 
rotation after warping the reference image by an estimated translation value ; 

(c) constructing a shift data/matrix that incorporat e s determining the image data shifts 
for the successive images with respect to reference each image; 

(d) modifying th e imag e shifts by multiplying th e imago shifts &i\ , s' m , &I' n and 



r e d e fining th e s e quantiti e s to incjlud e th e d e nominator factors - i — Q~ T z for points, 1 - Z" T z 
for int e nsiti e s and 1 - B > T for lin e s where s' m = q' w — q° and denotes th e imag e displacem e nt 
for th e m th track e d point, wh e re All * Q ^ e chang e in imag e int e nsity and wh e r e d e not e s 
th e * th int e nsity imag e and wh e re V n - V (p n ) denotes th e imag e int e nsity at th e n th pix e l 



position in f 1 * and whore SA 



from each image data shift; 



r e pr e s e nts th e lin e flow constructing a shift data representation 
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(e) calculating two rank 3 factor matric e s from th e modifi e d shift data matrix using 
SVD, on e rank 3 factor matim corr e sponding to th e 3D structur e and th e other rank 3 factor 
matrix corr e sponding to th e cam e ra motion; modifying the image shift representation bv 
multiplying the image shifts bv a predetermined factor which is dependent upon the image 
data, said image data being one more characteristics selected from a group consisting of 
points, lines and intensities: 

(f) r e cov e ring calculating first representation and a second representation from the 



modified shit data representation. 



said first representation corresponding to the 3D structure 



and said second representation co responding to the camera motion; from th e 3D structur e 



matrix by solving a lin e ar e quation; and 



(g) r e cov e ring reconstructmg the direction of the camera motion and 3D structure 
from the first and second representations , cam e ra motion matrix using th e r e cov e r e d 3D 
structur e . 



9. (Original) The method of claim 8, wherein the image data is one or more selected from 
the group consisting of points, lines and intensities. 



10. (Original) The method ofjplaim 8 5 wherein step (b) comprises: 

computing H and D c | , where H is a (N M - 3)x N tot matrix and D c// = C" I/2 D H T 

and Mot = 2N P + 2 N L + A^jphere N pt Afc, and N x equals the number of points, lines and 
intensities, respectively, aim C is a (Ni -1) x (Ni -1) matrix with C it = <5, r + 1 where C, r is a 
constant table of values fo|all images and<5 (7 , is an operator equal to 1 if / = V and 0 if / * /' 
and D = [S co L A 6J r A], H T us the transpose of the H matrix, where H is a matrix defined such 



10 



G:\NEC\1 l96\1370l\AMEND\13701.aml 



that H T is an identity maVix and annihilates ¥ x ,¥,, Y 2 where Y x r = frj fi> L ¥£ and 
similarly for the y and z components and whereby, and co L are constant weights that the user 



sets, and where ^ = 



M(xxA)}- 
{pl(xxA)}J' Ly ~ 



{p^-fyxA)}- 



( \i = r \\ where 4 is 



the unit normal to the plan® containing the line L and the camera center at the reference 
image, and where x, y, z ar| unit vectors in the x, y and z directions, and where 

where q=(x,y) is the image position of the 













{«»)}' 


X 


.{>?«}. 


5 T y 






{'"'(4. 



tracked point in the reference image and 



the three point rotational flows (x, y\ (x, y\ (x, y) are defined by 

r.' (,, .' < "]=[U,. ) ) •!'')•(;') 



and where 



¥ b = -{v/ r (l) (p)},^ = -j{w T (2) (p)^ b = -{w -r (3) (p)}, and where /> gives the image 

I 

coordinates of the pixel positions, and 



where A is an N f x IN J matrix where each row corresponds to a different image / and equals 

A, x (f x A/) and fx A, which we refer to respectively as the upper and lower directions, 
where SA} is the line flolv <5A \ = A) - A? , and z is the unit vector in the z direction and 
where 5 is a N. x 2N p matrix where each row corresponds to a different image / and equals 

[{y x } r \sy } T J , where - q° and denotes the image displacement for the m-th tracked 

11 
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point and where A is a N l x NL matrix, where each row corresponds to an image i and equals 

{a/' f where AI is the change V image intensity with respect to the reference image and 

where /' denotes the i-th intensity image and where V n = /' (p„ ) denotes the image intensity 

at the /i-th pixel position in /' and ^here the notation { V)\s used to denote a vector with 
elements given by the V a . 



1 1 . (Currently Amended) The method of claim 8, wherein step (d) comprises: 

computing the a^best rankp factorization of D CH « M^S^ T where are 
rank 3 matrices corresponding respectively to motion and structure, using SVD. 



12. (Original) The method of claim 8, wherein step (e) comprises: 

eliminating structure unknowns Q 2 , B z and Z" 1 from the Q> a to get ?>N tot linear constraints 

on the U and CI using thdflinear equation, [<D X O y 0 2 ] = H r S {3) U + fl 7 , W y % ]fl , where U 

and D are unknown 3x3fmatrices, and O and Y represent total translational and rotational 
flow vectors respectively, and solving these constraints with 0(N tot ) computations using the 



SVD, where the total translational flow vectors are defined by Q) x = 



X 



and similarly 



for the y and z components, and 
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"k^F 










.k^L 




ML 







where A is the normal to a first plane that 



passes through the center of projection and the imaged line in the reference image, and the 
normal to a second plane, B ispefined by requiring B • A = 0 and B • Q = - 1 for any point Q on 
the 3D line L, 

and where B u = B • P u and B L f B • Pjr are the upper and lower components of B, and where 



.MJ 



_ J w 



{?,!?;' } 



and where 



s -{^"X}^/v — ^"7j{^ = {z-^V/.p)}, and 

where Q is the 3d coordinate for a tracked 3D point in the reference image, and Z n is 

the depth from the cameraJo the 3D point imaged at the «-th pixel along the cameras optical 
axis; and // 

recovering the structure unknowns Q 2 , B z and Z~ l from 
[o, 0,0j=H r S ( %^ U and Q. 



13. (Original) The method of claim 8, wherein step (f) comprises: 
using S ( % » fe x oj and 

D c// * C"^ }<& r H T \c~^ {7; }<DjH T + C"^ {r z )O z r H T to recover the translations, and 



13 
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recovering the rotations \' x , co^ , co\ from 



»X +<^« +«X = -(c"%> r + fc}®; + ft Jof)^ , wherein C 



is a 



constant (Ni-1) x (Ni-1) matriAwith C ir = 6 U , + 1 and T represents the translation. 



14. (New) A computer system foilrecovering scene structure and camera motion direction 
from successive image data obtainegl from a multi-image sequence comprising: 
a processor; 

a video source whosl output is digitized into a pixel map by a digitizer, said 
output is sent in electronic form vi^Ja system bus for access by main memory; 
a display means; 

a user interaction rjSeans for selecting items on the display means; 

a storage device in communication with said processor, said storage device 
stores program code for programming said processor to perform a method comprising the 
steps of: 

(a) comparing a reference image to successive images, wherein the successive 
image is taken by translating or rotating the camera; 

(b) compensating for camera rotation after warping the reference image by a 
estimated translation value; 

(c) determining the image data shift for the successive images with respect to 
the reference image; 

(d) constructing a shift data representation from the image data shift; 



14 
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(e) modifying the image shift representation by multiplying the image shifts 
by a predetermined factor which is dependant upon the image data, said image data being one 
or more characteristics selected from a group consisting of points, lines and intensities; 

(f) calculating aflirst representation and a second representation from the 
modified shift data representati< n, said first representation corresponding to the 3D structure 
and said second representation < orresponding to the camera motion; 

(g) reconstructin * the direction of the camera motion and 3D structure from the 
first and second representations. 



15. (New) A computer-readafile medium that includes instructions for recovering 3D scene 
structure and camera motion direction from image data obtained from a multi-image sequence 
wherein said instructions, when executed by a processor, cause the processor to: 

(a) compare a reference image to successive images, wherein the successive 
image is taken by translating or rotating the camera; 

(b) compensate for camera rotation after warping the reference image by a 
estimated translation value 

(c) determine the image data shift for each successive image with respect to 
the reference image; 

(d) constrflct a shift data representation from the image data shift; 

(e) modifying the image shift representation by multiplying the image shifts 
by a predetermined factor which is dependant upon the image data, said image data being one 
or more characteristics selected from a group consisting of points, lines and intensities; 



15 
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(f) calculating a first representation and a second representation from the 
modified shift data representation, said first representation corresponding to the 3D structure 
and said second representation corresponding to the camera motion; 

(g) reconstructmg the direction of the camera motion and 3D structure from 
the first and second representations. 

16. (New) The method of Claim 1, wherein shift data representations comprise a first matrix 
corresponding to the 3D structure and a second matrix corresponding to the camera motion. 
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